Full-screen display with sub-display camera

ABSTRACT

An apparatus includes a camera, a primary display panel including a pixel array and an aperture adjacent the pixel array, an auxiliary display, and an optical assembly including a reflecting optical element and an actuator coupled to the reflecting optical element. The actuator is configured to switch the reflecting optical element between a first arrangement and a second arrangement. The first arrangement defines an optical path from the aperture to the camera and the second arrangement defines an optical path from the aperture to the auxiliary display.

FIELD

The disclosure relates to displays with integrated front-facing sensorsand mobile devices including the same.

BACKGROUND

Many mobile devices, such as smartphones and tablet computers, featurefront-facing sensors, i.e., sensors that face the user during use. Suchsensors include, for example, cameras (e.g., so-called “selfie”cameras), ambient light sensors, proximity sensors, finger printsensors, and IR sensors that can be used to detect 3D facial profiles.Other front-facing components include flood illuminators, speakers, andmicrophones, for example.

Another trend for such mobile devices includes shrinking the display'sbezel, the rim that holds the display and touch panel in place. In otherwords, there is a trend to have as much of the front of the device beoccupied by the display, where a so-called edge-to-edge display would beconsidered ideal. Because various front-facing sensors are often housedin the bezel, the trend towards smaller bezels have presented challengesfor device designers, who have opted for solutions such as including anotch in the display where the sensors are placed. However, including anotch means that not all the area within the bezel is available for thedisplay. Moreover, many people deem notches to be aestheticallyunsightly, and thus represent a non-optimal solution.

Positioning certain front facing sensors, e.g., cameras, behind thedisplay panel can be impractical because the display pixels and othercomponents degrade the quality of a wavefront passing through thedisplay, leading to blurry images.

SUMMARY

In general, in a first aspect, the disclosure features an apparatus thatincludes a camera; a primary display panel including a pixel array andan aperture adjacent the pixel array; an auxiliary display; and anoptical assembly including a reflecting optical element and an actuatorcoupled to the reflecting optical element, the actuator being configuredto switch the reflecting optical element between a first arrangement anda second arrangement. The first arrangement defines an optical path fromthe aperture to the camera and the second arrangement defines an opticalpath from the aperture to the auxiliary display.

Embodiments of the apparatus can include one or more of the followingfeatures. For example, the apparatus can include an electronic controlmodule in communication with the camera, the display panel, thesecondary display, and the actuator, the electronic control module beingprogrammed to operate the camera when the optical assembly is in thefirst arrangement and to operate the secondary display when the opticalassembly is in the second arrangement. The electronic control module isprogrammed to synchronize operation of the display panel and thesecondary display to present a continuous image across the aperture inthe display panel.

The display panel and the secondary display can have the same pixeldensity.

The display panel can be an OLED display panel.

The secondary display panel can be an OLED display.

The aperture can form a notch in the pixel array.

The reflecting optical element can be a prism including a reflectingsurface. The actuator can be configured to rotate the prism to change anorientation of the reflecting surface with respect to the aperture axis.

The actuator can be configured to switch the reflecting optical elementbetween one or more additional arrangements, wherein each of the one ormore additional arrangements define a corresponding optical path fromthe aperture different from the optical paths of the first and secondarrangements. Each of the optical paths of the additional arrangementsare from the aperture to a sensor or emitter.

The apparatus can be a mobile phone or a tablet computer.

In general, in another aspect, the disclosure features a method thatincludes: receiving light through an aperture in a primary displaypanel; directing the light, via a reflecting optical element, to acamera located behind the primary display panel; acquiring an image of ascene in front of the primary display using the camera; reorienting thereflecting optical element to receive light from an auxiliary displaylocated behind the primary display panel; and directing light from theauxiliary display panel, via the reflecting optical element, through theaperture, the light from the auxiliary display panel and light from theprimary display panel providing the imagery displayed on the device.

Implementations of the method can include one or more of the followingfeatures. For example, the reflecting optical element is reoriented byrotating the reflecting optical element.

The reflecting optical element can be reoriented in response to a cameraactivation from a user of the device.

Among other advantages, implementations feature full-screen displayswith front-facing sensors that do not include a notch. Rather, suchimplementations include an auxiliary display located beneath the primarydisplay and an actuable reflector to switch the light path between thefront-facing sensor and the auxiliary display.

The details of one or more implementations are set forth below. Otherfeatures and advantages will be apparent from the detailed description,the accompanying drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a plan view of an example mobile device featuring afull-screen display and a sub-display camera.

FIG. 1B is a cross-sectional view of a sub-display module of the mobiledevice shown in FIG. 1A. The module is arranged to direct light to thesub-display camera.

FIG. 2A is a plan view of the mobile device shown in FIG. 1A, with thesub-display module being arranged to direct light from an auxiliarydisplay to a user.

FIG. 2B is a cross-sectional view of the sub-display module of themobile device shown in FIG. 2A. The module is arranged to direct lightfrom the auxiliary display to the user.

FIG. 3A is a plan view of the mobile device shown in FIGS. 1A and 2A, inwhich the location of sub-display module and associated electroniccomponents is shown.

FIG. 3B is a schematic view of the sub-display module and associatedelectronic components.

FIG. 4 is a perspective view of another example of a sub-display modulecontaining multiple different sensors.

FIG. 5 is a schematic illustrating an example of a sub-display modulecapable of directing light along more than two optical paths.

Like references symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Mobile devices (e.g., mobile phones, tablet computers) are disclosedthat feature full-screen displays with a sub-display front-facing cameraand/or other front facing sensors or devices. The devices feature awindow in the display panel that permits the passage of light throughthe display panel in either direction. A sub-display module includes aflappable mirror or other actuable optic to direct light passing from orto the window along different paths. In one configuration (i.e., in an“imaging” or “selfie mode”), light passing through the window isdirected towards a camera. The user has the perception that he/she islooking at a camera though he/she is actually looking at the mirror. Inanother mode (i.e., in a “display mode”), the light path is to anauxiliary display contained in the sub-display module. The user has theperception that he/she is looking at the display though he/she isactually looking at the mirror. Such mobile devices can featurefull-screen displays with very small bezels.

Referring to FIGS. 1A and 1B, an example mobile phone 100 includes aprimary display panel 110 in a chassis that includes a bezel 112.Display panel 110 is an organic light emitting diode (OLED) display andincludes an array of light emitting pixels 111 supported by a substrate130. Display panel 110 also includes a window 120 located at a top edgeof the display. Window 120 provides an aperture through which light canpass to a sub-display optical module 200 located behind display panel110. The top surface of window 120 sits flush with the top surface ofdisplay panel 110, providing a seamless front surface to the user.

Optical module 200 includes an auxiliary display panel 210 and a camera220. Camera 220 includes a lens assembly 224 222 (e.g., including one ormore lens elements) and a sensor array 222 224 (e.g., a CMOS or CCDarray). A reflecting optic 240 is located between auxiliary display 210and camera 220. Reflecting optic 240 includes a reflective surface 232,which redirects light transmitted through window 120 as described inmore detail below. A rotary actuator 230 is coupled to reflecting prism240 and is configured to change the orientation of reflective surface232 with respect to window 120, thereby controlling the direction oflight transmitted by the window.

Generally, reflecting optic 240 can be any appropriate optic thatspecularly reflects incident light in the visible part of the spectrum.For example, reflecting optic 240 can be a prism with a mirrored surface(e.g., a surface coated with a reflecting material, such as silver oraluminum). Multiple layer optical reflectors can also be used. In someembodiments, MEMS devices, such as MEMS mirror arrays, can be used toredirect light to and from window 120.

Actuator 230 can be any appropriate rotary actuator capable of rotatingreflecting optic 240 back and forth between two positions. In manycases, actuator 230 is an electric actuator. In some embodiments,actuator 230 is a stepper actuator.

As depicted in FIGS. 1A and 1B, device 100 is in an imaging mode inwhich the front-facing camera, i.e., camera 220, is activated. In thisarrangement, rotating actuator 230 orients reflecting prism 240 so thatreflecting surface 232 directs light transmitted from the front of thedevice through window 120 towards camera 220. This allows camera 220 toacquire images of the scene in front of mobile device 100.

Referring to FIGS. 2A and 2B, when the front-facing camera is notengaged reflecting surface 232 is directed towards auxiliary display210. In this arrangement, reflecting surface 232 directs light emittedfrom the auxiliary display through window 120. Accordingly, a userlooking at primary display 110 sees the image display on the auxiliarydisplay in window 120. A controller, discussed below, coordinatesoperation of primary display 110 and auxiliary display 210 so that thedisplays collectively present the user with a seamless image.

In general, the size and shape of window 120 can vary as appropriate andshould be sufficiently large to provide an appropriate aperture for bothcamera 220 and auxiliary display panel 210. In some embodiments, window120 is square or rectangular with a diagonal dimension in a range from 1mm to 10 mm (e.g., about 1 mm to 5 mm, such as about 2 mm or about 3mm). Other shapes are also possible, such as other polygons, round,oval, polygons with rounded corners, and irregular shapes.

Furthermore, the auxiliary display panel can have a pixel density thatis the same as or different from the pixel density of the primarydisplay panel. In some embodiments, the auxiliary display features alower pixel density than the primary display. For example, where thewindow is located in a part of the primary display commonly used todisplay images that are lower resolution than, e.g., photos or movies,such as icons, the auxiliary display can have a lower pixel density thanthe primary display.

While the foregoing example features a window in the middle of the topedge of the primary display, more generally, the window can be locatedin any appropriate location of the primary display panel. For example,the window can be located at or close to a corner of the display. Insome embodiments, the window can be located at a long edge of thedisplay, at the bottom edge of the display, or away from an edge (e.g.,in which the window is surrounded on all sides by pixels of the primarydisplay).

Alternatively, or additionally, in some cases the optical modulecontaining the auxiliary display panel and camera can include additionaloptical elements. For example, in some cases, a module includesadditional optical elements, such as one or more lenses, in the opticalpath from the auxiliary display to the user. Such optical elements canform an image of the auxiliary display in the same plane as the primarydisplay.

Referring to FIGS. 3A and 3B, mobile device 100 includes an electroniccontrol module 300 that is electrically-connected to auxiliary display210, camera 220, and rotary actuator 230. Electronic control module 300is also in electrical communication with drive electronics for displaypanel 110. Control module 300 is programmed to coordinate and controloperation of the sub-display module 200 and primary display panel 110.For example, when a user activates imaging mode, control module 300sends control signals to actuator 230, causing the actuator to rotatethe reflecting surface 232 to face the camera 220, while at the sametime turning off auxiliary display 220 and activating camera 220.

When the user deactivates the imaging mode, control module 300 sendscontrol signals to actuator 230 to cause it to rotate reflecting surface232 in the opposite direction to face auxiliary display 210. Controlmodule 300 further coordinates operation of the primary and auxiliarydisplays

Various implementations of the subject matter described herein (e.g.,the electronic control module 300, primary display 110, auxiliarydisplay 210, camera 220 and/or any other component associated with themobile device and/or the primary display) can be implemented inconjunction with digital electronic circuitry, integrated circuitry,specially designed application specific integrated circuits (ASICs),computer hardware, firmware, software, and/or combinations thereof.These various implementations can be implemented in one or more computerprograms. These computer programs can be executable and/or interpretedon a programmable system. The programmable system can include at leastone programmable processor, which can have a special purpose or ageneral purpose. The at least one programmable processor can be coupledto a storage system, at least one input device, and at least one outputdevice. The at least one programmable processor can receive data andinstructions from, and can transmit data and instructions to, thestorage system, the at least one input device, and the at least oneoutput device.

These computer programs (also known as programs, software, softwareapplications or code) can include machine instructions for aprogrammable processor, and can be implemented in a high-levelprocedural and/or object-oriented programming language, and/or inassembly or machine language. As can be used herein, the term“machine-readable medium” can refer to any computer program product,apparatus and/or device (for example, magnetic discs, optical disks,memory, programmable logic devices (PLDs)) used to provide machineinstructions and/or data to a programmable processor, including amachine-readable medium that can receive machine instructions as amachine-readable signal. The term “machine-readable signal” can refer toany signal used to provide machine instructions and/or data to aprogrammable processor.

To provide for interaction with a user, primary display 110 can displaydata to a user. Camera 220 and/or other front-facing sensors can receivedata from the one or more users and/or the ambient environment.Controller 300 can thus operate based on user or other feedback, whichcan include sensory feedback, such as visual feedback, auditoryfeedback, tactile feedback, and any other feedback. To provide forinteraction with the user, other devices can also be provided, such as akeyboard, a mouse, a trackball, a joystick, and/or any other device. Theinput from the user can be received in any form, such as acoustic input,speech input, tactile input, or any other input.

While the foregoing example features an optical module that includes acamera and an auxiliary display, other arrangements are also possible.For example, in some embodiments, an optical module can include furthersensors behind display panel 110 in addition to a camera. Referring to 4a, an example of such an optical module 400 includes an auxiliarydisplay 410 along with a sensor module 420 that includes three sensors422, 424, and 426. An actuable reflector 440 that includes a reflectingsurface 432 (e.g., a mirror) provides an optical path from a window 401in a primary display panel (not shown) that switches between auxiliarydisplay 410 and sensor module 420. In embodiments where one or more ofthe sensors detect and/or emit non-visible wavelengths (e.g., IR sensorscan include infrared emitters and detectors), reflecting surface 432reflects both visible and non-visible (e.g., IR) radiation. In somecases, the different sensors can be integrated onto a common substrate.Alternatively, the module can be composed of separately packaged sensorsarranged side-by-side.

Of course, sensor module 420 is just an illustrative example. Generally,sensor modules can include any number of appropriate sensors (e.g., two,three, four or more) arranged in any appropriate configuration (e.g.,side-by-side, staggered, stacked or combinations thereof).

While the foregoing examples feature an actuable reflector that candirect light along two different optical paths, other configurations arepossible. For example, in some embodiments, a sub-display module candirectly light along three or more optical paths. Referring to FIG. 5,for example, a sub-display module 500 includes a light directing element(e.g., rotatable reflector) that directs light 541 transmitted by awindow through a primary display (neither are shown in FIG. 5) along oneof three different paths. In this example, two of the paths, 521 and523, lead to 2 sensors, 522 and 524, respectively. The third path, 511,directs light between an auxiliary display 510 and the window.

Although various implementations have been described above in detail,other modifications can be possible. For example, while the foregoingexamples use OLED displays for the primary and auxiliary display, it ispossible to use other appropriate display technologies. Moreover, theprimary and auxiliary display can use different types of displaytechnology, such as liquid crystal display (LCD) technology ormicro-light emitting diode (microLED) technology. For example, theprimary display can be a LCD while the auxiliary display is an OLED ormicro-LED display.

Other implementations are within the scope of the following claims.

What is claimed is:
 1. An apparatus, comprising: a camera; a primarydisplay panel comprising a first pixel array and an aperture adjacentthe first pixel array; an auxiliary display comprising a second pixelarray; and an optical assembly comprising a reflecting optical elementand an actuator coupled to the reflecting optical element, the actuatorbeing configured to switch the reflecting optical element between afirst arrangement and a second arrangement, wherein in the firstarrangement defines an optical path from the aperture to the camera andthe second arrangement defines an optical path from the aperture to theauxiliary display.
 2. The apparatus of claim 1, further comprising anelectronic control module in communication with the camera, the primarydisplay panel, the auxiliary display, and the actuator, the electroniccontrol module being programmed to operate the camera when the opticalassembly is in the first arrangement and to operate the auxiliarydisplay when the optical assembly is in the second arrangement.
 3. Theapparatus of claim 2, wherein the electronic control module isprogrammed to synchronize operation of the primary display panel and theauxiliary display to present a continuous image across the aperture inthe display panel.
 4. The apparatus of claim 1, wherein the primarydisplay panel and the auxiliary display have the same pixel density. 5.The apparatus of claim 1, wherein the primary display panel is an OLEDdisplay panel.
 6. The apparatus of claim 1, wherein the auxiliarydisplay is an OLED display.
 7. The apparatus of claim 1, wherein theaperture forms a notch in the pixel array.
 8. The apparatus of claim 1,wherein the reflecting optical element is a prism comprising areflecting surface.
 9. The apparatus of claim 8, wherein the actuator isconfigured to rotate the prism to change an orientation of thereflecting surface with respect to the aperture axis.
 10. The apparatusof claim 1, wherein the actuator is configured to switch the reflectingoptical element between one or more additional arrangements, whereineach of the one or more additional arrangements define a correspondingoptical path from the aperture different from the optical paths of thefirst and second arrangements.
 11. The apparatus of claim 10, whereineach of the optical paths of the additional arrangements are from theaperture to a sensor or emitter.
 12. The apparatus of claim 1, whereinthe apparatus is a mobile phone.
 13. The apparatus of claim 1, whereinthe apparatus is a tablet computer.
 14. A method, comprising: receivinglight through an aperture in a primary display panel; directing thelight, via a reflecting optical element, to a camera located behind theprimary display panel; acquiring an image of a scene in front of theprimary display using the camera; reorienting the reflecting opticalelement to receive light from an auxiliary display located behind theprimary display panel; and directing light from the auxiliary display,via the reflecting optical element, through the aperture, the light fromthe auxiliary display and light from the primary display panel providingan imagery displayed a device.
 15. The method of claim 14, wherein thereflecting optical element is reoriented by rotating the reflectingoptical element.
 16. The method of claim 14, wherein the reflectingoptical element is reoriented in response to a camera activation from auser of the device.